Tokenizer Namespace Reference

elementary tokenizer More...

Functions
char	block_delimiter (char entry_char)
	return the corresponding closing delimiter character, or 0 if entry_char is not a known delimiter
char	eat_space (std::istream &is, bool eat_line)
	skip space and new-line if eat_line==true, return the next character on the stream
char	get_character (std::istream &is, bool eat_line=false, bool eat_space=true)
	skip all character for which isspace() is true, and new-line if eat_line==true
std::string	get_symbol (std::istream &is, bool(*valid)(char))
	accumulate characters as long as valid() remains true
std::string	get_identifier (std::istream &is, bool eat_line=false)
	return next token that looks like a variable name More...
std::string	get_token (std::istream &is, bool eat_line=false)
	return next token More...
std::string	get_line (std::istream &is)
	accumulate characters until new-line is found
std::string	get_block (std::istream &is, char c_in, char c_out)
	read next block delimited by c_in and the corresponding closing delimiter More...
std::string	get_block_stripped (std::istream &is, char c_in)
	skip spaces and read a block delimited by c_in, or return empty string if c_in is not found More...
std::string	strip_block (std::string const &blok)
	remove enclosing parenthesis at the start and at the end of blok
std::string	get_until (std::istream &is, std::string what)
	read until what is found and stop immediately before (what is excluded from the returned string)
void	trim (std::string &str, const std::string &ws=" ")
	remove characters present in ws from the beggining and at the end of str
bool	get_integer (std::istream &, int &)
	read an integer More...
bool	get_integer (std::istream &, unsigned &)
	read an unsigned integer More...
bool	split_integer (std::string &str, unsigned &)
	try to interpret str as UINT sub. If successful, str is modified to be sub More...

Detailed Description

A Tokenizer is used cut a character stream into words

Function Documentation

std::string get_block	(	std::istream &	is,
		char	c,
		char	c_out
	)

This will read a block, assuming that character c_in has been read already. It will read characters until the corresponding closing delimiter is found.

std::string get_block_stripped	(	std::istream &	is,
		char	c_in
	)

This will skip spaces and new-lines until a character is found. If this character is equal to c_in, then the block is read and returned. Otherwise returns empty string "".

The delimiter parenthesis around the block are not included in the returned string!

std::string get_identifier	(	std::istream &	is,
		bool	eat_line = false
	)

get_identifier() reads words:

• starting with a alpha-character,
• followed by alphanum characters

bool get_integer	(	std::istream &	is,
		int &	what
	)

read an integer, or return false if that is not possible. The value of T will not change, if the input fails.

bool get_integer	(	std::istream &	is,
		unsigned &	what
	)

read an integer, or return false if that is not possible. The value of T will not change, if the input fails.

std::string get_token	(	std::istream &	is,
		bool	eat_line = false
	)

get_token() reads a block enclosed by '{}', '()' and '""', and returns it verbatim with the delimiting characters.

bool split_integer	(	std::string &	str,
		unsigned &	what
	)

split the string into an integer and the remaining string. The space after the integer is discarded.

Returns

success == true